fix: CLI crash + auto-generate YAML config templates from schema by viraatc · Pull Request #237 · mlcommons/endpoints

viraatc · 2026-04-01T21:56:51Z

Summary

Bug fix: CLI crash on --load-pattern + --target-qps (IndexError) — LoadPattern.type used alias= instead of name= on cyclopts.Parameter
Template generation: regenerate_templates.py auto-generates minimal + full YAML templates from Pydantic schema defaults
Pre-commit hook: auto-regenerates templates on schema/config changes; CI validates via --check mode

Templates

Two variants per mode (offline, online, concurrency):

_template.yaml — minimal: only required fields + <PLACEHOLDER eg: example> values
_template_full.yaml — all fields with schema defaults, inline # options: comments auto-discovered from Enum/Literal annotations, 2 dataset examples (perf + accuracy with parser columns), multiple endpoints

Init command

inference-endpoint init offline always generates the full template. Available types: offline, online, concurrency.

Tests

Hypothesis fuzz: 4000 random CLI flag combinations across offline/online/concurrency
Unit: all 8 templates (including eval/submission) validate via from_yaml_file (auto-discovered with glob)
Integration (e2e): every generated template is loaded, <> placeholders stripped to real values, and run end-to-end against the echo server — verifying templates produce functional configs
Init command: generates template + tested

Docs updated

AGENTS.md, CLI_QUICK_REFERENCE.md, DEVELOPMENT.md — all stale template/init references fixed

Test plan

python scripts/regenerate_templates.py --check passes
pytest tests/unit/ — 484 passed
pytest tests/integration/commands/test_benchmark_command.py::TestTemplateIntegration — 6 templates run e2e
Pre-commit hooks pass (ruff, mypy, prettier, template regeneration)

🤖 Generated with Claude Code

github-actions · 2026-04-01T21:57:02Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

Copilot

Pull request overview

Fixes a cyclopts CLI parsing crash triggered when --load-pattern is combined with load-pattern subfields like --target-qps / --concurrency in the online benchmark command.

Changes:

Annotates LoadPattern to adjust how cyclopts maps nested parameters (@cyclopts.Parameter(name="*")).
Updates the CLI parameter definition for LoadPattern.type to avoid the prior name collision.

Comments suppressed due to low confidence (1)

src/inference_endpoint/config/schema.py:360

This change is a regression fix for a CLI crash when combining --load-pattern with nested load-pattern fields (e.g. --target-qps). There’s existing automated test coverage for config validation in tests/unit/commands/test_benchmark.py, but no test currently exercises cyclopts parsing for this flag combination.

Add a regression test that parses benchmark online ... --load-pattern poisson --target-qps 100 (or directly parses OnlineBenchmarkConfig via cyclopts) and asserts it no longer raises and that config.settings.load_pattern.type/target_qps are set as expected.

@cyclopts.Parameter(name="*")
class LoadPattern(BaseModel):
    """Load pattern configuration.

    Different patterns use target_qps differently:
    - max_throughput: target_qps used for calculating total queries (offline, optional with default)
    - poisson: target_qps sets scheduler rate (online, required - validated)
    - concurrency: issue at fixed target_concurrency (online, required - validated)
    """

    model_config = ConfigDict(extra="forbid", frozen=True)

    type: Annotated[
        LoadPatternType,
        cyclopts.Parameter(name="--load-pattern", help="Load pattern type"),
    ] = LoadPatternType.MAX_THROUGHPUT
    target_qps: Annotated[
        float | None, cyclopts.Parameter(alias="--target-qps", help="Target QPS")
    ] = Field(None, gt=0)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/inference_endpoint/config/schema.py

gemini-code-assist

Code Review

This pull request modifies the LoadPattern class in the configuration schema by applying a class-level cyclopts.Parameter decorator and updating the type field's parameter definition to use the name argument instead of alias. I have no feedback to provide.

arekay-nv

Can we also add a test for this - seems like a change that shouldn't have gone in.

tests/integration/commands/test_cli.py

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.github/workflows/test.yml

tests/integration/commands/test_cli.py

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.github/workflows/test.yml

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.pre-commit-config.yaml

viraatc

duplicate

tests/integration/commands/test_cli.py

src/inference_endpoint/config/templates/offline_template.yaml

.github/workflows/test.yml

.pre-commit-config.yaml

viraatc · 2026-04-03T11:51:28Z

Review Council — Multi-AI Code Review

Reviewed by: Claude (Codex ran but produced investigation output, not structured findings) | Depth: standard

Found 3 issues across 3 files:

1 high (fixed)
1 medium (already fixed)
1 low (deferred)

#	File	Line	Severity	Category	Summary
1	`scripts/regenerate_templates.py`	95	high	error-handling	Pre-commit hook exited 0 on template generation failure — stale files could slip through. Fixed: now tracks failures and `sys.exit(1)`.
2	`.github/workflows/test.yml`	61	medium	security	Unpinned action SHAs in `schema-updated` job. Already fixed in latest push.
3	`tests/integration/commands/test_cli.py`	76	low	testing	`Optional` union types (`float

Also addressed all Copilot review comments (pinned SHAs, quoted pip install, heredoc for inline Python, expanded pre-commit files: regex, added except comment).

viraatc

added new schema-updated CI:

fuzz tests on CLI in CI
template validated against schema default in CI

NOTE: template now includes all supported fields

was pending items from past.
++ @rashid for thoughts?

tests/integration/commands/test_cli.py

src/inference_endpoint/config/templates/offline_template.yaml

.github/workflows/test.yml

.pre-commit-config.yaml

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/inference_endpoint/config/templates/concurrency_template.yaml

tests/integration/commands/test_cli.py

.github/workflows/test.yml

docs/DEVELOPMENT.md

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/regenerate_templates.py

src/inference_endpoint/main.py

src/inference_endpoint/commands/init.py

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/inference_endpoint/main.py

tests/unit/commands/test_benchmark.py

docs/CLI_QUICK_REFERENCE.md

Copilot

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docs/CLI_QUICK_REFERENCE.md

.pre-commit-config.yaml

Copilot

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/regenerate_templates.py

src/inference_endpoint/config/templates/online_template_full.yaml

src/inference_endpoint/config/templates/offline_template_full.yaml

src/inference_endpoint/config/templates/concurrency_template_full.yaml

Copilot

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/regenerate_templates.py

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/inference_endpoint/config/schema.py

scripts/regenerate_templates.py

tests/integration/commands/test_cli.py

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/inference_endpoint/config/schema.py

src/inference_endpoint/commands/init.py

src/inference_endpoint/main.py

Bug fix: - CLI crash on --load-pattern + --target-qps (IndexError) — LoadPattern.type used alias= instead of name= on cyclopts.Parameter Template generation: - regenerate_templates.py generates minimal + full YAML templates from Pydantic schema field defaults (no hardcoded values) - Full templates include inline # comments auto-discovered from Field(description=) and Enum/Literal annotations - Minimal templates: required fields + placeholders only - Pre-commit hook auto-regenerates on schema changes; CI validates via CI env var detection (check-only mode) - init command uses model_dump(exclude_none=True) for offline/online/ concurrency; falls back to handwritten templates for eval/submission Schema improvements: - Added Field(description=) to all HTTPClientConfig, AccuracyConfig fields - New ScorerMethod enum (pass_at_1, string_match, rouge, code_bench_scorer, shopify_category_f1) for AccuracyConfig.eval_method Tests: - Hypothesis fuzz: 4000 random CLI flag combinations - Unit: all templates validate via from_yaml_file (auto-discovered glob) - Integration: generated templates run e2e against echo server - Init command: all 5 types tested Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings April 1, 2026 21:56

viraatc requested a review from a team as a code owner April 1, 2026 21:56

github-actions bot requested review from arekay-nv and nvzhihanj April 1, 2026 21:57

Copilot started reviewing on behalf of viraatc April 1, 2026 21:57 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

src/inference_endpoint/config/schema.py Show resolved Hide resolved

gemini-code-assist bot reviewed Apr 1, 2026

View reviewed changes

arekay-nv approved these changes Apr 3, 2026

View reviewed changes

github-code-quality bot found potential problems Apr 3, 2026

View reviewed changes

tests/integration/commands/test_cli.py Fixed Show fixed Hide fixed

viraatc force-pushed the feat/viraatc-fix1 branch from 90fe9c8 to 80a79ef Compare April 3, 2026 10:56

Copilot AI review requested due to automatic review settings April 3, 2026 10:56

Copilot started reviewing on behalf of viraatc April 3, 2026 10:56 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

.github/workflows/test.yml Outdated Show resolved Hide resolved

.github/workflows/test.yml Outdated Show resolved Hide resolved

.github/workflows/test.yml Outdated Show resolved Hide resolved

github-code-quality bot found potential problems Apr 3, 2026

View reviewed changes

tests/integration/commands/test_cli.py Fixed Show fixed Hide fixed

Copilot AI review requested due to automatic review settings April 3, 2026 11:05

Copilot started reviewing on behalf of viraatc April 3, 2026 11:06 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

.github/workflows/test.yml Outdated Show resolved Hide resolved

.github/workflows/test.yml Outdated Show resolved Hide resolved

.github/workflows/test.yml Outdated Show resolved Hide resolved

Copilot AI review requested due to automatic review settings April 3, 2026 11:32

Copilot started reviewing on behalf of viraatc April 3, 2026 11:32 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

.pre-commit-config.yaml Outdated Show resolved Hide resolved

This comment was marked as duplicate.

Sign in to view

viraatc commented Apr 3, 2026

View reviewed changes

tests/integration/commands/test_cli.py Show resolved Hide resolved

src/inference_endpoint/config/templates/offline_template.yaml Outdated Show resolved Hide resolved

.github/workflows/test.yml Show resolved Hide resolved

.pre-commit-config.yaml Show resolved Hide resolved

Copilot AI review requested due to automatic review settings April 3, 2026 12:14

Copilot started reviewing on behalf of viraatc April 3, 2026 12:15 View session

viraatc force-pushed the feat/viraatc-fix1 branch from 8915750 to ffb87d9 Compare April 3, 2026 12:16

viraatc commented Apr 3, 2026

View reviewed changes

tests/integration/commands/test_cli.py Show resolved Hide resolved

src/inference_endpoint/config/templates/offline_template.yaml Show resolved Hide resolved

.github/workflows/test.yml Show resolved Hide resolved

.pre-commit-config.yaml Outdated Show resolved Hide resolved

Copilot AI reviewed Apr 3, 2026

View reviewed changes

viraatc force-pushed the feat/viraatc-fix1 branch from ffb87d9 to b781ff7 Compare April 3, 2026 12:21

Copilot started reviewing on behalf of viraatc April 8, 2026 22:14 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

scripts/regenerate_templates.py Outdated Show resolved Hide resolved

src/inference_endpoint/main.py Show resolved Hide resolved

src/inference_endpoint/commands/init.py Show resolved Hide resolved

Copilot AI review requested due to automatic review settings April 9, 2026 00:28

Copilot started reviewing on behalf of viraatc April 9, 2026 00:28 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

src/inference_endpoint/main.py Outdated Show resolved Hide resolved

tests/unit/commands/test_benchmark.py Show resolved Hide resolved

docs/CLI_QUICK_REFERENCE.md Outdated Show resolved Hide resolved

Copilot AI review requested due to automatic review settings April 9, 2026 19:14

Copilot started reviewing on behalf of viraatc April 9, 2026 19:15 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

docs/CLI_QUICK_REFERENCE.md Outdated Show resolved Hide resolved

.pre-commit-config.yaml Show resolved Hide resolved

Copilot AI review requested due to automatic review settings April 9, 2026 19:26

Copilot started reviewing on behalf of viraatc April 9, 2026 19:26 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings April 9, 2026 19:35

Copilot started reviewing on behalf of viraatc April 9, 2026 19:35 View session

viraatc force-pushed the feat/viraatc-fix1 branch from 4b2c8e7 to 93b8808 Compare April 9, 2026 19:37

Copilot AI reviewed Apr 9, 2026

View reviewed changes

scripts/regenerate_templates.py Show resolved Hide resolved

viraatc force-pushed the feat/viraatc-fix1 branch from 93b8808 to e916b74 Compare April 9, 2026 19:44

Copilot AI review requested due to automatic review settings April 9, 2026 19:48

viraatc force-pushed the feat/viraatc-fix1 branch from e916b74 to ea428c5 Compare April 9, 2026 19:48

Copilot started reviewing on behalf of viraatc April 9, 2026 19:49 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

src/inference_endpoint/config/schema.py Show resolved Hide resolved

scripts/regenerate_templates.py Show resolved Hide resolved

tests/integration/commands/test_cli.py Show resolved Hide resolved

viraatc force-pushed the feat/viraatc-fix1 branch 2 times, most recently from c4a1c6b to 302f591 Compare April 9, 2026 20:17

Copilot AI review requested due to automatic review settings April 10, 2026 00:06

viraatc force-pushed the feat/viraatc-fix1 branch from 302f591 to f030fd4 Compare April 10, 2026 00:06

Copilot started reviewing on behalf of viraatc April 10, 2026 00:06 View session

Copilot AI reviewed Apr 10, 2026

View reviewed changes

src/inference_endpoint/config/schema.py Show resolved Hide resolved

src/inference_endpoint/commands/init.py Show resolved Hide resolved

src/inference_endpoint/main.py Show resolved Hide resolved

viraatc force-pushed the feat/viraatc-fix1 branch from f030fd4 to 6bd97e2 Compare April 10, 2026 00:28

viraatc merged commit 4f746b6 into main Apr 10, 2026
7 checks passed

github-actions bot locked and limited conversation to collaborators Apr 10, 2026

Conversation

viraatc commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Templates

Init command

Tests

Docs updated

Test plan

Uh oh!

github-actions bot commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

arekay-nv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

This comment was marked as duplicate.

Uh oh!

viraatc left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

viraatc commented Apr 3, 2026

Review Council — Multi-AI Code Review

Uh oh!

viraatc left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

viraatc commented Apr 1, 2026 •

edited

Loading

github-actions bot commented Apr 1, 2026 •

edited

Loading

viraatc left a comment •

edited

Loading

viraatc left a comment •

edited

Loading